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WHAT IS CLAIMED IS: 

1 . A method for generating a scanner, the method comprising: 
receiving a definition of a plurality of patterns; 

receiving a definition of a respective association between each of the plurality 
of patterns and a respective executable action; and 

processing the plurality of patterns and the respective associations to form a 
scanner data structure capable of comparing input data to at least one of the plurality 
of patterns and causing execution of the associated executable action upon a match of 
the input data with the respective one of the plurality of patterns, the processing and 
the comparing being performed in a same active process. 

2. The method as recited in claim 1 wherein the scanner is capable of being used 
for at least one of a compiler front end, a syntax-highlighting editor, a text stream 
editor, a parser and a parser- filter. 

3 . The method as recited in claim 1 wherein the receiving the definition of the 
plurality of patterns is performed in the same active process. 

4. The method as recited in claim 1 wherein the plurality of patterns include 
regular expressions. 

5. The method as recited in claim 1 wherein the plurality of patterns include text 
patterns. 

6. The method as recited in claim 1 wherein the receiving the definition of the 
respective association is performed in the same active process. 

7. The method as recited in claim 1 wherein each respective association between 
each of the plurality of patterns and the respective executable action includes a 
pointer. 
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8. The method as recited in claim 1 wherein each respective association between 
each of the plurality of patterns and the respective executable action includes an index 
number. 

9. The method as recited in claim 1 wherein at least one of the respective 
executable action includes a respective action object. 

10. The method as recited in claim 1 wherein at least a portion of at least one of 
the respective executable action is generated in the same active process. 

1 1 . The method as recited in claim 1 wherein at least one of the respective 
executable action includes respective program code. 

12. The method as recited in claim 1 wherein at least one of the respective 
executable action includes respective data. 

13. The method as recited in claim 1 wherein the processing includes forming a 
scanner definition string from the plurality of patterns and respective associations. 

14. The method as recited in claim 13 wherein each respective association is 
represented in the scanner definition string by a respective indicator. 

15. The method as recited in claim 14 wherein the processing further includes 
saving a mapping between each respective indicator and a respective action-pointer. 

16. The method as recited in claim 14 wherein each respective indicator includes 
a respective index. 

17. The method as recited in claim 14 wherein each respective indicator includes 
a respective a number representing a respective pointer. 
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18. The method as recited in claim 13 wherein the scanner definition string has a 
format of a scanner definition file of a scanner generator, 

19. The method as recited in claim 18 wherein the scanner generator is a Flex 
scanner generator. 

20. The method as recited in claim 1 8 wherein the processing further includes 
inputting the scanner definition string into a scanner generator core modified for 
processing the scanner definition string so as to form a processed scanner definition 
data structure. 

21 . The method as recited in claim 20 wherein the scanner generator core is a Flex 
scanner generator core. 

22. The method as recited in claim 20 wherein the processing further includes 
converting the processed scanner definition data structure into the scanner data 
structure. 

23. The method as recited in claim 1 wherein the scanner data structure includes a 
respective indicator representing the respective association between each of the 
plurality of patterns and the respective executable action. 

24. The method as recited in claim 23 wherein each respective indicator includes 
an index. 

25. The method as recited in claim 23 wherein each respective indicator includes 
a pointer. 

26. The method as recited in claim 13 wherein the processing further includes 
saving the scanner definition string and the scanner data structure. 
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27. The method as recited in claim 26 wherein the saving is performed to a 
persistent memory device. 

28. The method as recited in claim 26 wherein the processing further includes 
saving a mapping between the scanner definition string and the scanner data structure, 

29. The method as recited in claim 28 wherein the saving is performed to a 
persistent memory device. 

30. The method as recited in claim 28 further comprising: 
receiving a definition of a plurality of second patterns; 

receiving a definition of a respective second association between each of the 
plurality of second patterns and a respective second executable action; 

processing the plurality of second patterns and the respective second 
associations so as to determine a second scanner data structure capable of comparing 
the input data to each of the plurality of second patterns and causing execution of the 
respective second executable action upon a match of the input data with a one of the 
plurality of second patterns, the processing each of the plurality of second patterns 
and the respective second association and the comparing the input data to each of the 
plurality of second patterns being performed in a same second active process; 

wherein the processing each of the plurality of second patterns and the 
respective second association includes: 

forming a second scanner definition string from the plurality of second 
patterns and respective second associations; and 

comparing the second scanner definition string to the saved scanner 
definition string and loading the saved scanner data structure as the second scanner 
data structure when the second scanner definition string matches the saved scanner 
definition string. 

3 1 . The method as recited in claim 30 wherein the saved scanner definition string 
and the second scanner definition string have a format of a scanner definition file of a 
scanner generator. 
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32. The method as recited in claim 3 1 wherein the scanner generator is a Flex 
scanner generator. 

33. The method as recited in claim 30 wherein the loading the saved scanner data 
structure as the second scanner data structure is performed using the mapping 
between the scanner definition string and the scanner data structure. 

34. The method as recited in claim 30 further comprising inputting the second 
scanner definition string into a scanner generator core so as to form a processed 
scanner definition data structure when the second scanner definition string does not 
match the saved scanner definition string. 

35. The method as recited in claim 34 wherein the scanner generator core is a Flex 
scanner generator core. 

36. The method as recited in claim 34 wherein the processing further includes 
converting the processed scanner definition data structure into the second scanner 
data structure. 

37. The method as recited in claim 30 wherein the processing further includes 
saving the second scanner definition string and the second scanner data structure. 

38. The method as recited in claim 37 wherein the processing further includes 
saving a mapping between the second scanner definition string and the second 
scanner data structure. 

39. The method as recited in claim 1 further comprising: 

associating at least one respective start state of a plurality of start states with 
each of the plurality of patterns; and 

setting a current start state, the current start state being one of the plurality of 
start states; 
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wherein the processing includes processing the at least one respective start 
state along with each associated pattern so as to form a part of the scanner data 
structure, the comparing being performed so as to compare only the patterns of the 
plurality of patterns associated with a respective start state equal to the current start 
state. 

40. The method as recited in claim 39 wherein the setting the current start state is 
performed so as to reset the current start state to another one of the plurality of start 
states at least once during the comparing. 

41. The method as recited in claim 40 wherein the current start state is reset using 
at least one of the respective executable action. 

42. The method as recited in claim 40 further comprising maintaining a stack of 
the plurality of start states, the current start state being the top of the stack. 

43. The method as recited in claim 39 further comprising associating a respective 
context with each of the plurality of start states for controlling the respective start 
state. 

44. The method as recited in claim 43 wherein each context includes at least one 
respective rule defining a start and an end of the context. 

45. The method as recited in claim 39 further comprising maintaining a stack of 
the plurality of start states, the current start state being the top of the stack, each of the 
context for controlling a position of the respective start state in the stack. 

46. The method as recited in claim 1 wherein the input data includes a data 
stream. 

47. A method for scanning input data, the method comprising: 

receiving a definition of a plurality of patterns; 
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receiving a definition of a respective association between each of the plurality 
of patterns and a respective executable action; 

processing the plurality of patterns and the respective associations so as form 
a scanner data structure; 

comparing the input data to the a plurality of patterns using the scanner data 
structure; and 

when the comparing results in a matched one of the a plurality of patterns, 
executing the respective executable action associated with the matched pattern; 

wherein the processing and the comparing are performed in a same active 
process, 

48. The method as recited in claim 47 wherein the scanning is performed in at 
least one of a compiler front end, a syntax-highlighting editor, a text stream editor, a 
parser and a parser-filter. 

49. The method as recited in claim 47 further comprising receiving the definition 
of the plurality of patterns in the same active process. 

50. The method as recited in claim 47 wherein the receiving a definition of the 
respective association between each of the plurality of patterns and the respective 
executable action is performed in the same active process. 

5 1 . The method as recited in claim 47 wherein at least one of the respective 
executable action includes a respective action object. 

52. The method as recited in claim 47 wherein at least a portion of at least one of 
the respective executable action is generated in the same active process. 

53. The method as recited in claim 47 wherein the processing includes forming a 
scanner definition string from the plurality of patterns and respective associations. 
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54. The method as recited in claim 47 wherein the processing further includes 
inputting the scanner definition string into a scanner generator core modified for 
processing the scanner definition string so as to form a processed scanner definition 
data structure. 

55. The method as recited in claim 54 wherein the processing further includes 
converting the processed scanner definition data structure into the scanner data 
structure. 

56. The method as recited in claim 53 wherein the processing further includes 
saving the scanner definition string and the scanner data structure. 

57. The method as recited in claim 56 wherein the processing further includes 
saving a mapping between the scanner definition string and the scanner data structure. 

58. The method as recited in claim 57 further comprising: 
receiving a definition of a plurality of second patterns; 

receiving a definition of a respective second association between each of the 
plurality of second patterns and a respective second executable action; 

processing the plurality of second patterns and the respective second 
associations so as to determine a second scanner data structure capable of comparing 
the input data to each of the plurality of second patterns and causing execution of the 
respective second executable action upon a match of the input data with a one of the 
plurality of second patterns, the processing each of the plurality of second patterns 
and the respective second association and the comparing the input data to each of the 
plurality of second patterns being performed in a same second active process; 

wherein the processing each of the plurality of second patterns and the 

respective second association includes: 

forming a second scanner definition string from the plurality of second 

patterns and respective second associations; and 

comparing the second scanner definition string to the saved scanner 

definition string and loading the saved scanner data structure as the second scanner 
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data structure when the second scanner definition string matches the saved scanner 
definition string. 

59. The method as recited in claim 58 wherein the loading the saved scanner data 
structure as the second scanner data structure is performed using the mapping 
between the scanner definition string and the scanner data structure. 

60. The method as recited in claim 58 further comprising inputting the second 
scanner definition string into a scanner generator core so as to form a processed 
scanner definition data structure when the second scanner definition string does not 
match the saved scanner definition string. 

61 . The method as recited in claim 47 further comprising: 

associating at least one respective start state of a plurality of start states with 
each of the plurality of patterns; and 

setting a current start state, the current start state being one of the plurality of 
start states; 

wherein the processing includes processing the at least one respective start 
state along with each associated pattern so as to form a part of the scanner data 
structure, the comparing being performed so as to compare only the patterns of the 
plurality of patterns associated with a respective start state equal to the current start 
state. 

62. The method as recited in claim 61 further comprising associating a respective 
context with each of the plurality of start states for controlling the respective start 
state. 

63. The method as recited in claim 62 wherein each respective context includes at 
least one rule defining a start and an end of the context. 

64. A scanner comprising: 
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a scanner data structure including processed information for a plurality of 
patterns and respective indicators for associating a respective executable action with 
each of the a plurality of patterns, the scanner data structure being capable of being 
used to compare input data to the plurality of patterns and cause execution of the 
respective executable action upon a match of the input data with a one of the a 
plurality of patterns, the scanner data structure being formed and the comparing being 
performed in a same active process. 

65. The scanner as recited in claim 64 wherein the scanner is capable of being 
used for at least one of a compiler front end, a syntax-highlighting editor, a text 
stream editor, a parser and a parser-filter. 

66. The scanner as recited in claim 64 wherein a definition of the plurality of 
patterns is received in the same active process. 

67. The scanner as recited in claim 64 wherein a definition of each of the 
respective indicator is received in the same active process. 

68. The scanner as recited in claim 64 wherein at least one of the respective 
executable action includes a respective action object. 

69. The scanner as recited in claim 64 wherein at least a portion of at least one of 
the respective executable action is generated in the same active process. 

70. The scanner as recited in claim 64 wherein the scanner definition data 
structure is formed by a process including forming a scanner definition string from 
the plurality of patterns and respective associations. 

71 . The scanner as recited in claim 70 wherein the process further includes 
inputting the scanner definition string into a scanner generator core modified for 
processing the scanner definition string. 
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72. A computer readable medium having stored thereon computer executable 
process steps operative to perform a method for generating a scanner, the method 
comprising: 

defining a plurality of patterns; 

defining a respective association between each of the plurality of patterns and 
a respective executable action; and 

processing the plurality of patterns and the respective associations so as to 
form a scanner data structure capable of comparing input data to each of the plurality 
of patterns and causing execution of the respective executable action upon a match of 
the input data with a one of the plurality of patterns, the processing and the comparing 
being performed in a same active process. 

73. The computer readable medium as recited in claim 72 wherein the scanner is 
capable of being used for at least one of a compiler front end, a syntax-highlighting 
editor, a text stream editor, a parser and a parser-filter. 

74. The computer readable medium as recited in claim 72 wherein the method 
further comprises defining the plurality of patterns in the same active process. 

75. The computer readable medium as recited in claim 72 wherein the defining 
the respective association between each of the plurality of patterns and the respective 
executable action is performed in the same active process. 

76. The computer readable medium as recited in claim 72 wherein at least one of 
the respective executable action includes a respective action object. 

77. The computer readable medium as recited in claim 72 wherein at least a 
portion of at least one of the respective executable action is generated in the same 
active process. 
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78. The computer readable medium as recited in claim 72 wherein the processing 
includes forming a scanner definition string from the plurality of patterns and 
respective associations. 

79. The computer readable medium as recited in claim 72 wherein the processing 
further includes inputting the scanner definition string into a scanner generator core 
modified for processing the scanner definition string so as to form a processed 
scanner definition data structure. 

80. The computer readable medium as recited in claim 79 wherein the processing 
further includes converting the processed scanner definition data structure into the 
scanner data structure. 

8 1 . The computer readable medium as recited in claim 78 wherein the processing 
further includes saving the scanner definition string and the scanner data structure. 

82. The computer readable medium as recited in claim 8 1 wherein the processing 
further includes saving a mapping between the scanner definition string and the 
scanner data structure. 

83. The computer readable medium as recited in claim 82 further comprising: 
defining a plurality of second patterns; 

defining a respective second association between each of the plurality of 
second patterns and a respective second executable action; 

processing the plurality of second patterns and the respective second 
associations so as to determine a second scanner data structure capable of comparing 
the input data to each of the plurality of second patterns and causing execution of the 
respective second executable action upon a match of the input data with a one of the 
plurality of second patterns, the processing each of the plurality of second patterns 
and the respective second association and the comparing the input data to each of the 
plurality of second patterns being performed in a same second active process; 
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wherein the processing each of the plurality of second patterns and the 
respective second association includes: 

forming a second scanner definition string from the plurality of second 
patterns and respective second associations; and 

comparing the second scanner definition string to the saved scanner 
definition string and loading the saved scanner data structure as the second scanner 
data structure when the second scanner definition string matches the saved scanner 
definition string. 

84. The computer readable medium as recited in claim 83 wherein the loading the 
saved scanner data structure as the second scanner data structure is performed using 
the mapping between the scanner definition string and the scanner data structure. 

85. The computer readable medium as recited in claim 83 further comprising 
inputting the second scanner definition string into a scanner generator core so as to 
form a processed scanner definition data structure when the second scanner definition 
string does not match the saved scanner definition string. 

86. The computer readable medium as recited in claim 72 further comprising: 
associating at least one respective start state of a plurality of start states with 

each of the plurality of patterns; and 

setting a current start state, the current start state being one of the plurality of 
start states; 

wherein the processing includes processing the at least one respective start 
state along with each associated pattern so as to form a part of the scanner data 
structure, the comparing being performed so as to compare only the patterns of the 
plurality of patterns associated with a respective start state equal to the current start 
state. 

87. The computer readable medium as recited in claim 86 further comprising 
associating a respective context with each of the plurality of start states for controlling 
the respective start state. 
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88. The computer readable medium as recited in claim 87 wherein each respective 
context includes at least one rule defining a start and an end of the context. 
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